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The purpose of this paper is* .to. shat'e some of the techniques 

evolved to evaluate a deve.loping school-community change 

; ' . ■ ' ' ' \ 

process. Work-' on this process is now being done by the 

ft ^ 

staff of the Northwest- Regional Educational Laboratory's^ 
Rural Education Program under a contract with* the National 
Institute of Education. The process being developed is 
oa.lled .the Rural Futures Developtnent (RFD) Strategy and is 
aimed a^Nhelp^g rural school systems and;their constituents 
initiate and carry out changes in their schools. . ^ ' \ 

This paper is not injtended to describe the RFD StY^ategy in 
detail. 'However, for orientation purposes' we will jprovide 
a brief overview of the Strategy in 'the first section. In 

, o * 

Section II we will describe briefly the ,thre,e stages of 
Strategy testing b^ing employed'. ^ Ii) t^ie third a*nd. major \ ' 
section we will discUss the variety, djf^ evaluation techniques 

employed durang initial •iiesting of one component of the 

' ' ' ' '^'^ 

Strategy. And in the fina^l section we will present implici- 

- ' ■ . ' < , ^ / 

tions this' in^itial testing experience has' for. t^ose who /nust 
evaluate other .school-community change' processes * 

, • ' Section I - What is B'eing Evaluated 

The RFD Strategy has as i.ts goal 'improved local problem 
solving "through (1) increased control of ed.ucational change 
by iQcal rura]^ community members and school personnel and - 
(2) improved support services for r^ural schools by state * 
and regional education agencies. ^ ^ * - * 



To initiate use of the RFD, Strategy , .a state education agency 

must»-agree to employ the ' Strategy in a pilot site" in the' 

^tate/ This site is chosen -from^regional educational service 

agencies and local school diatricts that indicate their will- 

lingness to participate in^ the Strategy. 
^ • .* 

* ' \ 

Selected ^individuals from the^regiorial agency that is chosen 

are then trained to/become "process facilitators" in pilot , 

communities and schools in the districts they 'serve. In 

• ' • ' . . / 

.a.ddition^ othei^ personnel in the state and regional agencies 

are given training that will/prepare them to support 'the 
. * ' / \ » 

/ \ * 

process in the pilot 'site arid .to\expand it to othe^r sites. ^ 
Fpr example, cerl/ain state /agency \employ^s^, through; an 
apprenticeship, are tr^in^ to be futur:^^ facilitators 

After some ini;tial training, proce^ss facilitators engage 
local citizens, educators, studentls and school board members 
-'^in'^a comprehensive process for planning , school change. ^ In 
thi^s. process , local people identify a priority problem to be 
SO Ved select the solution from kmong others that -have been 
prdJpoVed, implement the s^JLect^d/ solution Ipind evaluate its 
effectiveness in solving *the problem. 

■ - \ ' \ 

The groups, primarily involved /in carrying out this process 
at the 16cal level are- (l)^h4 school staff, (2) a school- 
community group (made up of /representative educators, 
d'ommuhity members and studentfe), (3) the school board and 
.(U)* school leaders; ^ * /, , ■ . - * 



The Rural Education Program is presently cjeve^loping materials 

and procedures for use by ccmiail^ity members., school ^staff ' 

jn.embers, school boaVd member's , ^school leaders, process 

facilitators and support agency personnel' to implement the . 

ito) Strategy. At present, the Strategy is in its second 

testing phas'e (discussed in the next section) • This testing 

differs from previous testing in that, for the first time, 

it invol*ves installing the entire Strategy in one western 

«* , * \ ■* 

* • ^ ' .... 

state. Information gathe1?ed through this test is critical 

k « 

to further refinement ' of the Strategy. , * * 

RFD Strategy Objectives ^ • , ^ 

The RFP Strategy objectives can be divided into two. general 
categor^ies — :process objectives and.c^utcpme objectives. Pro- 
cess objectives focus on 'the events that occur when 'the RFD 
Strategy is. implemented appropriately • Examples of process 
events are ^Formal and informal training sessions for process 
facilitators, meetings of school-community groups, agreenvents 
on budget allocations • by school\board,s , and events in which 
certain systematic problem .solving ^procedures ' are used in 
educational agencies. jDutg^bme objectives, describe changes 
that should take ^place^j>^thg end of ^ two-year imp'lemerlta-, 
tio^ effort. These outcome objectives ' include : (1) improved 
cpmAiunication about decision making among school* personnel, 
students, school board members, and community members, (2) 
greater influence by students', sch661 staff , and community 

' - • . ^ . ..." 4 >^ 

members upon decisions 'made by the school-board ^d staff,. 



(3) ' increased commitmenf on the part of state and regional 

education agencies to thie use of proce'dures which promote 

local* problem solving and decision making and (4J) increased 

state' and /regional ability to^ support local problem solving y 

activities. Thus, process -objectives focus on means, and 

outcome /ob'je'ctives focus on ends. 'Refinement of these 

ob-jectijves eontinyes as Strategy testing continues. ^ , 
• / . ^ • 

I ' ' ' - * 

Basic /Assumptions Underlying the RFD Strategy 

I •. * ■ * ^ * ' . 

The*RFD Strategy is built on several ba?ic assumptions about 

eff.^ctive educational^ change : . (1) people who are affected 

by /decisions should help make those decisions, (2) communi- 

ti;4s and schools should leai^n how to select, from among 

'several alternatives, the solutions they deem best to solve 

^heir own problems , (3) a systematic approach to solving 

/problems helps people make the best use of Available time 

and resources and ^4) certain skil^-S are needed by partici- 

pants in decision making to ensure that problems are ^ 

satisfactorily resolved. The RFD Strategy cioes not attempt ^ 

* * . ' « * 

to provide packaged solutions to rural problems. Rather, 

/ ^ * .... ~^^r^ 

it provides a process that encourages local initiative anc 

participation in determining educational ^change . 

■ ^ I ■■■ ■ ■ ■ ' ' ■' '.. 

Theoretical Base of the Strategy , " 

/The theoretical base upon which the change strategy is built 

/ " ' * * 

is -derived primarily from the work of Havelock (1969), 

. - ° ■ . 

Lippett (1973), Bales (1970), Williamson (19 71) , aifd 'Schijiuck 
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and Runkel (1972t. Essential elements *are': (1) establishing 
a linkage system for diffusing innovations,'^ la Hivelock; 

(2) providing an external change agent--the process fac;Ll- 
itator — to Intervene in the system, as suggested by Lippett;- 

(3) providing training in 'communicatioiH, problem solving and 
group grooess as described .by Schmuck, Runkel atnd Bales; and 
(U) developing "^he school's organizational capacity 'to inquire 
into Its own functionji^ng various levels,, a la' Williamson.^ 
The basic learning model upon which the Stra^tegy is built is , 
presented in the work of Woodruff >(1971)^. Fcrr a furtl^ei?* , 
explication of the theoretical base ^ ".seffe Sayers (197.U). 

. Section tl - tjow'the Strategy 
Is Beine Developed and Tested'" 

t)evelopn!ent . ' 

' ,* ♦ 

An iterative and interactive process of Strategy development 
•and testing is being used by the RuraJI^ducat ion Program. ^ > 

Such a procedure is built oa the program's* strong commitment $ 
^to f ield based develo^itient . | 

Initial development work on tjie RFD Strategy was done by 

'IB. 

having teams of developers w6rk rather independently on major 
product's and processes that should constitute the Strategy.'* 
For example, one group worked on th,e products and processes 
needed to establish a 'schools-community group* that would 
conduct an assessment of Educational needs in the local 



community. Other .developers* have focused their ^ork on the 

3 



products and processes needed to help the ^school oi^ganization 
incorporate dea^ired changes (identified through the needs 

^ -4 . 

assessment) into the educational system. Other developers 
focused on developing products or processes t6 help state and 
retgional educational service agencies to provide res-ponsive* 
support to local educational agencies. 

Thfe -^developer tealns producjed initial .drafts of materials for 
conununity members , school personnel , schCol board members^ 

^: process facilitators and support agency personnel to use in 

implementing the Strategy. They also developed plans for" 
training process facilitator.s and other RFD ' personnel. 
During the development of such initial drafts of products ^ 
id;^p^lans, the first phase of^ testing occurred. 



The First Phase^^e^E^ Testing 

The first phase of testing — exp^ox^atory testing — is primarily 
-'intended to obtain user and/or . expert rea^rtiQns to products 
and product pieces as' initial drafts are' developed .^^"^^Foj 
* expert or user review, the ^product must be in a tangible ' 
form and b^' accompanied by clear statements of its ^goals , 
objectives, and intended uses. Prior to such a review, 
limited trials of the product^ with intendeid users may also * - 
be carried out to Answer selected development cfUestions. 
Thus, separate el^ements of the Strategy were 'tested with 
experts and users before the entire Strategy wafe installed 
. in a -f^ield site. ' ' . 



io 



During exploratory testing, questions such as the following 

r' 

we're aske'd: (1) What skills and -knowledge are not adequately 
covered, by the product?. 02) What changes in product format, 
methodology and communication systems "should hk made? (3) 
Are Strategy and product objectives adequate? (5). What 
problems'. are likely to be encountered during implementation 
of the Strategy? (6) What is the cost of product or process 
use? ' ^ * . • 

The Second Phase of Testing ' ' * * 



Before embarking on the second phase of. development and test- 
ipg,. develo^per teams were reconstituted so they no longer > 
focused on organizations and institutions (i.e., community,' 
school and support agencies).' The new teams work across - * 
these areaq in ways that provide . greater opportunity, for 
seeing where gaps and redundancies exivSt in the products and ^ 
processes constituting the Strategy.' For eTiam^le, one te^^m ^ 
focuses. o^i procedures for Qonducting all training sessions, * 
one focuses on manuals process fabilitators will use in 
workifig with both the community and school and another is 
actively engaged in installing the total Strategy' in a site. 
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TotaX^S4:,5at^g^^ constitute^s the basis for the 

second testing phase. This phase, referred to as site A 



testing ^ isjnow in progress.^ It inv6lves one state education 
agency and one -local education .^eney' or school , district 
Site A tresting is primarily de^'^signed to 4etermin'^ where' gaps 
or red*indan.cies ih.the^ Strategy occur^' and*whe^hel? qr not 
. ■ %i. „ • \ 7 

'• J ' • : ' ■ • •• . > \ ' 



\ 



* ..^ • ^ . 

intended processes cAn be implemented r These determinations 
» • *^ ? 

are based on how parts functipn in the totals context of 
Strategy use. Thus, in Site A., "evaluators are almost exclu- 
sivelv concerneid with . providing' data that facilitate -refine- 
ment and delineation- of the Strategy definition and proca^s 
objectives . - •/ - . 



The Third Phase of Testing 



For the third phase of testing, the Stra-fegy, its cpmponbnt 
products\and processes are intended ta^j^ in a close~to~f inal 
state. Thi^ testing phase is being^ referred to as Site B 
testing-. It, -the Site A ^testing," w^ll involve ins-tailing 

- <- '-^ ;■• . .'1 - ^ . ' • 

the entire Str'ategy in a site. But m.Site-B, ins^tallation 
will be* on a broader - and more complete scale than it was in 
Site ''A/' It is planned that Site B will consist, of a state 
education agency, at least tvo' regional educational service 
agencies,' at least four local education agencies and at. \ ^ ^ 
lea'st two comparison 'local education agencies. 'This pha.se ^ 
of testing is intended vto; begin witliin the next nine months. . 
GBecause \of -the length of time required to complete each af ^ 
* these phases of Strategy testingr-^about 6tw|| year?-T-there.is 
considerabl'e overlap of the testing phase s^) ^ . ^ 



'in the Site B test, the evaluators will Be concerned not 
only with tl^e degree to which the Strategy ia .implemented 
(as reflect^ed in the- process objective's), Iht also- with ttxe 
extent' to which the Strategy's outcome objectives are'metT 
An .evaluation design *for Site' B involving comparison • sites 
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is planned to increase the^ ^'validity , of the e\Aa3Axation and 

" ' - ^-v • • ' . • • ■- ^-T'^. ■/■ 

thus make it 'possible to more predibly attribute .observed* 
effe'fits. to thye RFD ;$f ratfegy^ ' . 

TKe Purpose .of Testing ^ -^j./^, - -.'.v^V. ■ ^ ' 

The -purpose o!f each ph*ase of testing is strictly to-provide- ■ 
Strategy develppers arrd\p.nstallers-. wil:h formative' evaluation 
•data which will help them bef'itie.the Strategy. The sequence 

^ ' - ' , • ! 

of exploratory testing,-' Site A testing ,^ an?i Site B testing 
is designecj^ to*' provide evaluative data appropriate for the 
particular state of the.RFD Sti*^tegy's development'.*.. 

' , Section III^ Techniques 'Employea ^ . V- 

During ^Exp'toratory Testing . ' y 

■ . ■ . \ • 

' i A discus/s522ru^&/>the exploratory testing of all parts .pf the 
r J?FD Strategy is beyond the scope of^this paper., There^ore^ 

% twe will restrict our discus^sion to the exploratory -te^t of 
^. \/ one part of ^he Strategy. Interest in community in volvement^ 
in 'schools, appears to be. growing rapidly. If this is true, 
it is very important' t-hat gvaluation procedures fdr^assessing 
* such activities exi3t; Good, pertin^rit evaluation can maxi- 
mize the. lihood that any community involvement will. be 

. ii • - ^ • • - ' 

of value ro both the community and the ^school. - Fop th^s.* 
reasbn we have chosen 'to focus, on the^ evaluation of that » ' 

. . , - • . . ^ . • / • - / 

aspect of -tWe^-RFD ^Strategy that^is concerned with .obtaining 

<v ■* " ' * " ^ ' ' ^ • 

the GomJTrurrity ' a^i^nvolverrient in instituting e.ducatipnal change, 
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During the exploratory testing phase of th^ RFD Strategy's ^ 
development f ^the community aspectrof the 'Stipategy was the ^ - 
mo^t*^ extensiv^y tested part- The evaluation work consti- 
tuting this test was designed primarily to provide developers 
with ihformaition that would help them improve ' their materials 
and training procedures- However, a.s^ond purpose of testing 
was to help ervaluators determine which ,kinds of evaluation 
techniques woitld be most useful for the subsequent .testing 
phases. " * " . * ^ 

During the exploratory testing .pljase the* maj or trial of the . 
community aspect .Of the ^•I'D Strategy >ccurred over -ah 18- • 
month peHodV'*^F'irty membeijs of a stat-e ecti©a±ian agency 
were grained as process facilitators to •e^stablTish s}^ol-^^- 
community groups* whose purpose 'was to conduct a needs* \^ 
assessment and. work for the implementation of educational 

• • • ^7 

change based bn-that needs assessment. The tra,ining- was 
done ^through seven ^formal two- or thr^e-day trainin^^^^s^s 
conducted by'' developers.. 'Concurrently, about 7 5^^rcent .of 
the trainees engaged in field 'work.- As pTOcess- facilitatprs , ^ 
they assisted" .rural^ communjLties in detei^mining their local 
educ^ational needs and establishing" means to m'e'et 'such need^^- 

The evaluation techniques' that are discussed below wfere usejj/ 
to gather information fipom (1) process facil'itator trainees 
duri'ng forjirifil trai*ning sessions and while working in communi- 
ties, (2) trainers of process facilitators (^3 ) community 
members where the process was being used and (4) documents 



/ 

/ ■- 



\ - \ : \ ' j y ' : . • 

■ ■ -'• " V ■ ^ . ' ' ■ ■ 

which cpntairied data about . the >^f^roee'&s -The evaluation • 
, .'techniques employed and discussed "beTow ar^; s]?:ill self- ' 
r.. rating scales', training sessioxi quest iojiii^ire s , .training 

' ■■ ' , fi"' ■ ■ ~ • . ■ 

observation schedules, lags of field events, school- community 
membejn questionnaires and interview^ , 'tjrocess events pheck- 
lists and i?evi?ews of l^iptorical do9Uii>erits' (slich as nevjspaper' 
articles , memos an'd^t;?ip reportr'sy. An. overview of 'each'./ * 
technique /S^howing what question's were asJdr,efes^5 the nature 
of the results obtained and the -i;;elative mer^^ each* 
technique can be found in Table 1.. .-^^ 

'In the following discussion , 'we will jpro vide fe-xamples 'of the 
use of/each of the s ^* 1:^ cliques . . Since this pape5:>.'*is 'direct'ed 

toward'' eyaluator^- of .co'mmuni^y-sch'ool change rarocesses,- // 

' / \ V \ ' ^' ^ ' * ^ ^ . ' . -^^ 

^i?aft)ier thart^^tKo'se tdiiiecelSly inv'^olyed in, cdnducting such = . 

.processes , ^theVdispu^g'ion provides a minimal .amount p^^de.:tail 

. , /, coVicerriin&^specific.daia, gath^ed. Rather, it. focuses on the 

^* Strengths And weaknesses, of vari&us evalua"bion techniques for 

' • tW* exploratory jte^'lling ^-pf the'*^bommimdty aspe.ct of tjie' RFD 




/Strategy. ^ .V . v , . ' V ^ 



-^*V:' V ' ' } ^r^^lnihg Sess'lonj:Bv^luat-i6n..Techniques" N ^ 



formal training sessions were 



4 . : , I. As mentioned earjliier> seven forir 

:M1, V'V%:* V' ' ihe^ly f'ozp prd.cess jfacilitaiof^s. During' tKe training. ses*sipns 
'L^^-^-^'V thlReeist:j!f.pes of ©v^-luation techniques ^ere *used : ^ (£0 observa- 
* j ti|^n ^^cnedules , .( 2^)i^,skil3L self-rating^- and C3) questionnaireb 
•0^/A:'ri'-;^2' fojeiiV'^k 'oh sat isfaLVtipn.- with sessi'or)S>'' ' 



^ipn.'with ---^---'^ — ^i^*'' • • ' 
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Nonpatti'cipant and participant tra'/iping session • 

T ^ ' */ ~ 

obseyvatibn^ schedule s . To provide^ 'developers with data that 

Qoulc^elp them improve the training /Sessions , it w^s first 

hece^jj^y. to .-'determine if training ^as actually occur ing a^^' 

' planned. ^ Certain components- of trailing were to 'be, conducted 

'as seminars and others as exercises. Non participant obs'erva- 

"tion schedules were used to determine whe*th^er -or not tjrainers 

actually conducted trainirig according to the methodolo'^y 

intended. Two. observers trained'' in the distinctive features 

of seminar and ex'ercis-e methodolqgles recorded betiaviors at 

•each session. , . * ' ; ' 

During sessions a participant observation: s*chedule vas *used 
,.to determine as' specifically as^possible the.atrong and weak "> 
areas of triaining: Two trainees were go^ven an qbservation 

- • ; / • . 

schedule^and asked to record their reactions to the training. 

They were also asked-lfo 'give a brief description of what was 

happening at five-minute intervals during the training session 

» • 
so that evaluatoas could, 'd^te^mine when trainees became bored 

or confused by what .was presented or done. Particulajply 

c^lear or interestii^g- .points could also , be detected in this 

way. Developers found'" this precise micro Level data- from 

both the nonparticipaht and participant observation. proce- 

diires to be helpful in (a) clarifying distinctions between. 

the modes of training^ and (b) detiermining s^pecifiq strong 

and weak parts of the training • 



Skills self-rating 'scales * Skills self-rating scales, 
were used at various points 'during the -^eries of training' 
sessions t'o determine, if trainees, felt that they were 
acquiring ,the skills being' taught . The training wa'S ' intended 
to provide trainees'- with skills in such areas as, ihterpe^- 

^ s % . 

sonal conununication 5 problem solving and'helpihg other-s to ^ 

«» ' * ^ • 

locate resources. Self-ratings of trainee jself-confidence 

" » ... * 

in their skills were used to measure skills acquisition. 

One such self -rating sc^ale was administered at the fourth 
training session on a p're-post basis. The scales were 
developed by evaluators following discussions w'ifh. ti^ainers 
^concerning the critical sKill-s to be learned by tu^-inees. ' * 
T-rainees were classified as having high^ low or;no field . 
experience when they responded to the rating s,cales. Appen- 
dix A contains th^ rating scales and the average "responses 
•for the jthrpe groups cOn both the. pre- and the post-session 
ratings. Such data showing pre-post t%raining differences 
between trainees with va1?ying degrees of * experience were 
useful to developers as they decided which aspects qf the 
Jraining needed strengthening". The data were also' helpful 
to evaluators as they worked on refinjing skills self-ra1|ii;ig' 
scales for. future testing activities*. - \ 

To determine what cor>tent^. might have been overlooked in 
designing ^the training, we acquired from other experts in 

V ^ , , ' ^.-^^ ' ^ . • ^ ■ ' 

the field a bro^d sample of self-rating knowledge and &fclll 

iteims dealing with the same general areas as those taught 
\ / , ' ' '^^ 

^n/ the workshop. Such -items were used to construfct 'another 

' ' . ^ -'^ . / 

19. ' \^ 



self-ra^tiag'' instrument (see Appendix B) on which trainees 
were asjked- to rate. Irheir - levels of HnOwle^ge or skill and 
to indicate whe:feher op not they knew where to get help with 
weak areas. This served as a check on whether* the training 
had adequ'ately covered 'a content area* ^ , * » 

A self-rating sca[le can be helpful in evaluating many kinds ' 

■ . '.■■■/ 
o'f training, and the data obtained .can be quickly ' analyze^d* ' / 

" . ^ . . . • / 

Since a self-rating scale requires clear statements of the / 
skills to be acquired, its preparation can also serve a 
* ^useful function ifi helping^ trainers clarify their intents 

for the 'training session.* Finally, self -ratings can be^ used, 
. in conjunction with pee^ "and trainer ratings to produce an 
I'^even iriore reliable estimate of the trainees* abilities. 



T^rainee ques'tionnaires . Another technique used during 

' ^ ^ • ' '■■'../ 

exploratory testing of tra^ini^g wa^ the administration to 



'^ration t 

trainees of questionnaires using Likert scales; open-ended 



-►questions, and multiple choice response items. For exaijiple, 
tirainees were asked' to respond to a short- ten-item form after 
each one-hour training event. Each trainee responded to one 
of tw|o forms'* of the questionnaire so that more data could be - 
gathered 'without over-burdening respondents. This' technique 
was' used to gather data on the technical quality of mat6;ri^ls, 
contribution of matrerials to -learning, degree 'to which event^ 
maintained trainee interest:, clarity of concept develbpment, 

• appropriatenes'^s . of the level of detail presented ^nd adequacy 

• of opportunity ^ for discussion. For .example ,^ trainees were 

20 ■' - 16 ' \ 
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- ^ 

aske4 if they preferred more ^ less or the same 'amount of 
reading Tnaterial , practice ^ in new behavior, ^rycturing of- 
•work, discussion and^ theoretical framework. * (See Appendix C 
for an efcample of one'^such questionnaire.). The data gathered • 
were very helpful in giving' developers id^eas pn how to** improve^ 
the training sessions. - On the basis of such data 'in conjunc- 
tion .with data gained through the other means discuss'ed above, 
revisions were made 'in the training methodology , the format 

and content of materials, and the * communication system used 

i * 

to convey the content of training. I^ultiple methods of data 

i 

•gathering were extremely useful in providing th^ scope and 
depth of evvaiuation necessary for this .complex training 
pro gram 

f 

Community Event Evaluation Technique 

During exploratory testing of the grommuni;ty aspect of \the RFD 
Strategy, the , community changey^ooess was 'actually conducted 
in several sites in ^lorth ceniiral Washington and in Montana 
by process facilitators who were receiving trainir^g. All* of 
the communities were isolated and small ^n4 had predominantly 
agricultural, mining or recreation based economies. To track 
the implementation ^of the prcj^cess, evaluators employed three 
^ techniques : itf^ field events logs, ( 2 J ^process * events check- 
lists and historical document review^ and (3) school- 
community member questionnaires and interviews. 

^ ^ Field events log . Two! process facilitators were^asked 
to \k^ep a regular log of allVfield activfties. After* every 



commuErity activity in which they participated, the ♦process 
facilitators completed a debriefing form (see Appendix D). 
and put it in their log- Periodical-ly these logs were sent ^ 
back to the Northwest Regional Educational Laboratory XNWREL) 

< ' ' , V 

for synthesis. The field events log wa^ useful m providing 

a derailed description of the change process a$ it was imple- 

mented in three communities * ' The constant reporting allowed 

process facilitators tb note their immediate .needs for mate- 

rials or 'further training, as well ^as process . activities 

» * . ' 'V 

which worked well or poorly. Further discussion of the field' 

reports by NWREL developers and active process facilitators • 

helped' developers understand and inc9.rporate this ^data wKen^ 

revising the change iSrocess and the manuals ^and materials . 

used-^in implementing the process. Recording these kinds of 

data was time consuming for proQess facilitators and for those 

who synthesized the data, into a narra^tive account. Although^ 

the data were useful, they were costly to gat^ier .and analyze. 

In' addition, their accuracy was highly odepend^r\t' on tl;ie i 

process facilitators' perceptions.,' TJiis^ "micro^ \?iew of the 



proce'ss, however, was^ very useful in providing examples of 
"how to do it" and "how not to do it" for inclusion in 
process facilitator manuals. ^ ^ 

Process events -checklist and review of ^I^ig-tOrical ^ 



documents . The second technique used was a jproces^, events 
che.cklist, -the purpose of which was to det.^i^'mine if anS when 
the^events identified as critical to' the community aspect, of 
the Strategy had occurred in communities engaged/. in tjife 1 



process^. To confirm the dates and occurrence , of events, 
checklist data wfere. supplemented by dafa from historical 
documents. T)ie process eventa checklist provided a.*"macro" 
•view of the proj(j^ss . The checklists were- Constructed based 
on the community change model, which identified critical 
events that should occur in the community. ' These* checklists 
were filled out by>'evaluators , based^on discussions with 
process facilitators and 5?eview of historical documents. 
The- historical documents reviewed include^ newspaper, articles , 
^meeting -minutes , letters, memos, and trip report recd^ds . 
Snbh documents were found to be usefuT, but often not compre- 
htensive, systematic, or detailed er^o.ugh to provide all the 
information needed t© document that critical events had 

I 

Occurred. The completion of the checklist allowed determina- 
t|hOn^, of^%ow frequently a particular event occurred and seipved, 
along ^,wit}i' other data collected, fo help S^yelppers determine. 



Although li.ttl*e ,sV,stematic analysis of historical documents 
was conducted during exploratory ^testing,* e valuators are now 
jusing a procedure fdr-' monitoring news^^^per articles in -the 
Site>A e^valuation based ^on^^a procedure considered during 
exp]JoTatory, testing. ^,JFhi^ procedure is pf^imari,ly. an adap- 
tation of a .techn.ique"lised l^y Morris and Guenter (1973^).. The^, 
proce<iure "is designed 4:0 obtain p. mea'Surfe pf hoth th^ quantity 
and the quality of newspaper ^ticles ^ abput local education . 
activities. %he techriique ^in\^lves categorizing^^arlricles 
into vari9us cOnt^ent .ar^eas , describing the ,t6ne 6^ articles 
and simply^*counting column .inches covering educational issues. 

, ■ ^ r- rl . ■ 

Anpi^her technique that shpw^, pEom4.ae for future evaluations 
is content "analysis of school board ml-nutes used .hsy'Mi'chel 
(1973). * T X'^^-' . ' ' / 




wftich events wejre most critica;!' ta the change process and,. 

which^ were redundant or optional.- Thdse data allowed 

J ' ' ' - ' . . . • . ' , 

developers to e].iminate, add or conso^Lidate events to a^rrive 

at a'more refined description' of the process. Xhds'kind of ^ 

overview of the process, also allowed' developerjs to sketch 

timelines V for implementing the, change pr<^e.ss. These data 

indicated that the process , will .take more J:han the anticipate^ 

two years to\ implement . Based on these data,' developers are 

now attempting to streamline the process to fit a two-year 

time frame. Costs were also cap.culated based^ on the time 

requir'ed-'to implement the process and the salaries and - 

expense^s budgeted for one process facilitator serving 'four 

Communities. 

School-community group membery and administrator 

quest ionnaires and followup interviews > The third technique 

— ' I * . 

used to evaluate community events was the administration of 
a questionnaire^ and followup intervi-eV to school-gommunity i . 
group (SCG) members and administrators who parti ciliated in the 
change process. * The questionnaire consisted of Liiker"{: scale 
items, open-ended questions, and multiple choice response 
questions. Questions were asked about adequacy of group ^ 
re^esent^ion, attendance at ^meetings , decisions made by 
the group, major agcompiishmeht of the .group, skills^ learned, 
resource-s used, timeymoney spent on the process, SCG/school 
.board relationships, cons,tancy of group membership, communi- ^ 
c^rtitjn adequacy, and -flegt'ee of influence the SCG had on 

' • ^ .24 



^ichool decisions. (S^/.Appehdix'' E for a copy.' of this 
instrument . ) • . ' 

Four school-community gr^up , members and. four school 
administrators wer^ mailed quest^ionnaires . ' A^l^er complet-ing 
the questioanaares 5 'respondents met with evaiuators for a 
general discussion. This was foll'oVed by injdivijjual inter- 
views on 'pnly those question^^ that were answered incompletely 
or in an un/slear- manner. this procedure ensured comple.te r and 
understandable responses for. aid iteihs.. 

This technique-- of u^ing a questionnaire' with a fol'lowup - ^ 
/interview proved to be well worth the^e^tra t-im^ involved' 
*since it^^ proyid-ed clear, complete responses from the small 
number 6f respondents. This techi^ique provi'ded data on 
impor4;ant suQc'ess indicatgrs and provided formative feedback 
th^t couTd be useful in modifying- the process .and .its accom- 
panying - products • Ic^ was ,also particularly helpfiul to 

evaiuators in providing initial data* from which to build 

. ^ . * * - w, • 

more structured questioiinaires for future use. 

The variety of techniques^ used in, evaluating community events 
provided both a .micro and a macro view of the process. The 
combination of these views allowed the' evaluation 'audience 
to better determiine what was or was not occuring and how to 
change the process and products so .that they might produce 
improved results*. It is very unlikely that one technique • 
alone could provide the scone and. depth of data required to 
make these i'mportant^ determinatipns for a complex process. 

. . ' dit^* - , 21 



^ * section IV - Implications of Testing 

. for Other. School -Community Change Proc^'sses ^ 

As mentioned earlier, .the pi^mary piSrpose of. the evaluation 
work^ during Exploratory testing was to provide developers 
with information thatf^Jbuld hel^p them^ prepare prototype 
drafts of products and defiije important elements of the 

Strategy. For evaluators there was another p^urpos^^:' "^o" 

< — ♦ 
determine the evaluation technicjues and^instruments that 

woulp be most useful for subsequent Strategy tests. 

, Based on. exploratory testing ^^fefjeriences , evaluators have' 
made certain decisions related' to the future phases of 
testing. We feel tljat these decisions could ai^so have 
.significance for evaluators planning to test 'other school- 
conununity change processes. . ^ ^ 

One of the major decisions ^malN^ is that it is essential to 
use a variety of e\/aluation techniques.. The RFD Strategy is 
of such*^ complexity that no one technique is adequate'^ to 
determine 'the strengths and weaknesses of the Strategy. 
Questionnaires^, onsite interviews meeting observation 
'S'chedules, logs^, checklists, review of historical documents 
and reviews of work (e.^., needs assessment questionnaire's 
and reports) by community and school groups are all planned' 
^r use in future Strategy testjing. Sucn a variety is 
necessary to prov itle^ a iQ.ng^ and a^ short-range, view^ of thev 
process* and a.mic^o and macro view of events. 
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., ■ ■ ■• 

,A second major ^ecisiion is that it is critical for evaiu^tion 
• * inst'ruments *tb build as much as possible on already- existing 
• ins'truments It is very ti,me consuming to develop the high 
-quality instruments which are critical to doing- a "^credible 
evaluation. One way to save time is to make use of ,,the work ^ 
done by other eva;Luatars and researchers working^on related 

» projects. Tfce instrument development in which we are pres- 

. . ^ ' - '-^^^ ' . y A '~ ' 

'Pritly engaged, ^as preparation for future testing, involves 

extensive ,revaew of instruments used by otheij, organizations 

'Jthat are involved.in measuring' school and community change. 

To facilitaire suoh .work, evaluators of RFD have gathered , and 

organi.ze"d a file' of available instruments in the areas, .of ^ 

communityVschool change and related 'skills . These instruments 

along with those dfscusse.d in- this paper are being used, as' 

\ 1 ' \ • ' ' ' 

* -they appear apppopriaite , to 'deVelop more refined and^ valid 



instruments for future testing of the RFD Strategy. Instru- 
ment preparation also ini^lves v^extensive reviej; by ^valuators 
within the Ruyal Education Pragram and NWREL .for technical 
quality and p'r;btect ion- of hiiman subjects. 



A th^rd de.cision is to emphasize the importance of interB^ction 
with developers of the' Strategy to ensure that objectives :are^/^ 
clearly stated, instruments are focused on the. most critical^ 

' questions to 'be answered and the extent bf structure of the 

, - r> 

instrument's appropriate for the- information so\ighl . -&uch 

. ' - . ' ' ' ^ ^ • '\ ' 

interaction is deemed critical , not only ^o help eviLluators 

determine prioritie^s ^and procedures for data collection, but 

-also to^help developers articulate dnd identify the criti^cal 

' ' ^ \ ' - ' \^ ' ' • - . 2 3 
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. skills to be communicated and the processes to be used. 

\ 

Given that our evaluation is strictly formativ^e throughout 
all three phases of 'testing, such a dual purpose for this 
activity is especially important. 
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• ;. ' . APPE^^DIX'B■ • ' 

EVALUATION S,KILl(S S'ELF-RAT.ING SCALE • ' ! " 

a. ■ ■ • ■ 

• i: - . • . "* 

. DIRECTIONS . ••• •• . 

In responding to this instrument, you are asked to do two t^jj.ngsr- first, 
react to a checklist of 32 items W indicating the level of competence 
^ou feel you possess' in each specified area; second, indicate whether or 
not you know where to seek assistance in each area.' ; 

In, using the chedklist, please keep in mind the following interpretations^ 
"for each of the' three competency (ability or knowledge), categories you 
.might choose: 

' High Degree o'f" Knowledge or Ability 

This category refers to a level of competence that enables you to 
'proceed independently or ta,Ke primary" responsibility for an activity 
in the area. ' ^ 

. . Moderate Knowledge or Ability 

\ ' * » - ' ' '* . 

This category. refers to a Tevel of competence less than that in the 
category above J but sufficS^ent to-enabile you to communicate 
intelligently about the area or be a team member 'on an activity in 
the area. ^ ' '. - ' 

Almost No Knowledge or Abjjity • „ ' ' 

— ■ 1 • ■ . - 

\ This category refers to a level of competence insufficient to be . 
classified in either of the first' two categories. 

In responding to each of the'32 items, pfease mgke two checkmarks - tffie. In 
^a column to the left of the ifem,. and one in a column t6 the right of the 

item^ . ' ^ B-l 

• f - 

♦ Northwest Regional Educational Laboratory (^ffj}\% This Instrument meets NWREL.gufdelines 
710S.W. Second ftvenucPortland.Oregon i Aw-^s I for the Protection of Human Subjects. 




.^97204 Telephoae(503) 224-3650 

^^gmamSSf Rural Futures Devetepment(RFD) Strategy 
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• APPENDIX C < • - 

, STAGE V AND VI TRAINING EVENT FEEDBACK FORM 

Numb,er E vent Number. Trainer ; 

The materials will assist my work with, the profc-ess : ^ 

A great deal 5 H 3 2 , . 1 . Not at. all 

ThG trainers were weli-organizedl for the event: 

A great deal 5 - 4 3 2 ' 1 " ' Notj^at all ^ 

The expected outcome of this event was: 

Very clear . • 5 4 3 -2—1 Very .unclear 

To what extent was the expected outcome reached for you? 
Not at all 1-^.2 3 H .5, Completely 

The conten^^f this event was appropriate to my needs^ as 
a Process ^cilitator: 
Strongly agree 5 H 3; /^'2 



1 ^ Strongly disagree 



The trainer (s) were very effective in helping me *to lea3?n 
what I wanted to from this event: ' - J 

Strongly disagree 1 2 ^3 H 5 Strongly agree 



My interest in the session Was: 
Very higli 5 4 3 ' 



'2 % ^ 



•1 ' Minimal 



F,or future -training of Process Facilitators this event 
should' be: ' . ^ ^ , 

Required . 5 ^ 4' 3 2 . 1 ^ Cropped 



The • event was :* 
Too trainer 
dominated S 



7* * 



2 ' 



Too. 'trainee 
dominated 



Please ch^ck the "box that best describes what you, recommend 
be done in futures workshops for the jFollowing aspects of 
this event: ' * ' - ^' 



. Same 



More 



Less 



A. Reading Material 






» • 'J 


B. Structuring oT the 
Work 








Q» Amount of^ Practice in 
J^ew Behavior* 








D. Amount of Discussion 








E. , Theory: Whj^;to Do, It, 

— r-: 









7)6 S.W. Second ft veauc PoftlandjOregon 
97204 Telcphoft^03) 224 -3650 




for the Pfotecrion.of Human Subjects. <^ 
Rural Futures DcvebpmentCRFD)Stfamgy' 
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stage 

Event 

•Date ^ 

Common ity 
PF's ^ 



) 



, AoDenclx. D' 
CHANGE PROCESS DEBRIEF 



(Please Print) 



Recorder 

Time Used (in hours) 
Field Prep._ 



Check Appropriate BoxCes): 
' Field Notes-^ I 1 ^ 
Interview. * i | 
Observation | I 



No. of SCG in 
' Attendance 



!}" 



not filled out by a PF 



Travel 



EVENT INFORMATION 


ACTIVITY- TITLE: 
/AC/ . . , 






WHAT HAPPENED: 

;do/ 


(What did you do; what did they do)^ . 





(attach other sheets if necessary) 



WHY DID YOU 00 WHAT YOU DID THE WAY yOU-OID (What variables influenced your 
/WH/ • action): 



HOW SUCCESSFUL WAS THI,5 EVENT (Give indicators): 
/SC/ 



I 



HOW COULD IT BE IMPROVEUs- 



MATERIALS INFORMATION 



MATERIALS U5EU (Give titles, or other descriptors): 
' /MU/ 



CHANGES YOU WOULD SUGGE.ST OR THAT YOU MA.tfE (Please at'tSch a copy of any 
/MY/ • . revts^ed' material): , 



MATERIALS NEEDED BUT NOf AVAJCABLE (Descrite): 
./MX/ • ' . . . 



TRAINING INFORMATION 



SKILLS YOU FEEL YOU USED DURING THE E'Jm (That a new PF ought to be 
/KL/ ^ ' ' ' ~ trained in): 



/ 



S.K^LS YOU nFeDED, BUT1)IDN'T HAVE : 

1w ~ 



COMMENTS: 
/AB/. 



PLANS MADEf 
/FR/ 



ERLC /\ 



S8 . 
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APPENDIX E 



SURVEY OF THE BREWSTER SCHOOL, COMMUNITY GROUP - SCAT 



o^l^ 



Please respond in a concise'and complete manner to the following questions. 
We are most interested, in your feelings and opinions. Thank you. 



I am a: 



1. 



b. 



staff member * \ ' • ' 

student 

community member 

administrator 

Kqw well does the SCG represent t|ie ideas of all factions in 
your community? (Circle one) 



Not at all 
representative 



1 2 3 4 5 
What groups, if any, .are not represented by the SCG? 



Very representative- 
all groups have a voice 



Rate each. gr6up (i.e., students, staff, community) individually on 
the dimensions listed below. (AlV groups can have the same rating 
if that is the way you think it is.) 



.a. 



b. 



C. 



Did they attend regularly during the first six 
months of SCG meetings? (Give them a rating 
from 1 to 5 where 1 means "seldom if ever 
attend meetings'* ^'efnd 5 means ^"attend almost 
.^afll n)ee tings".) 

^Have they attended regularly during the last 
six months? (Rate from 1 to 5 as in item a 
above. ) 



How acti\)e are they in the SCG group? (Rate 
from'l/to 5 where 1 mea.hs "very inactive" and 
5 means "very active"^ 



d.. How interested was each group in the SCG? 
, (Rate each from 1 to 5'Where 1 means "very 
^uninterested" and 5 means "very interested".) 

K 

* How ihfluential .was each group in SCG planning 
'v, "^and decisions? (Rate each from 1 to S where 
1 means "have very little influence" and 5 
means "very influential"^ ) * > 






• 


/ 






/ 





















Northwest Regional Educational Laboratory r . 7x^1 Ih'S if)strument meets NWREL guidelines 

710 S.W. Second ftvenue Portland.Oregof^ ; \ 1 for the Protection of Hu/narvSubjects 
97204Telephon(>(503)224-3650 39,;^VVw^J ' / r^l 

>aCTrtfm:V Rural Futyres Devclofifnent'^gFDjrStrate^y^ 



Approximately how of ten v^ere^SCG meetings held? (Check one) 

(1) once a week - " r 

(2) ' twice a month . ^ . 

(3) once a month ; 

(4) once every two months ^ * ' * - 

(5> I other ('^^se specify) 

Wa% there- enough time to accomplish tasks?=^S_ NOJ 



Approximately how often were TFT meetings held? (Check one) 

(Ir) once a v/eek' " ^ , ^ ' ^ 

(2) twice a montf> 

(3) once a mqnth . ' 

(4) . once .every two months 

(5) other (please specify) > - 

Did you have any prolonged recess of SCG activities? YES » 
NO Why? . ' , 



Why did people attend the meetings? 



Why didn't people attend the meetings? 



How pften did j^eo^le other than SCG members attend^msetings? 
(Circle^ne) • ^ . ^ . . » 

Never 1 2 3 4 5 Always • \ 



If so, can you categorize the^e people in terms of groups they\ 
m^ight represent in the coirmunity? \ * 



Whatr are the major, goals or purposesl^of the SCG? 



b. What are the tasks or functions wh4ch the SCG should be 
carrying out? ^ . , , ' , 



] 

0 I 



c. What are" some decisions that.,lg;^e SCG has- made? 



d. What. are the major accfompl tshments of the SCG up to this point 

^ir\ time? , . ■ i ' ■ , \ 

_ . , • ■ -' 11,. 



Please -list the top concerns that your community identified in jfche 
ne/ds assessment survey and briefly describe what has been done (if„ 
anything) in relation to each concern: , ■* 

• Needs/Concerns . Action Taken to fiate 



b." ' - • • ; 

^ ^ — ^- 



d. 



a. What resources (i.e.*, libraries,' local agricuTtural extension 
. agent, etc.)' in your . community have you used in your work with 
.-the SCG? ^ > ' ' 



b. What resources' '{lie. , State Department of Education, Senator, 
etc.)* outside .of your community have ybu used? ^ 



a*^ .Approximately how much money has your district spent to support 
'SCG activities? ' ^ 



^"'^ . E-3 



b; Approximately how ma(ny -hours per' week do yQu spend on SC6' 
related activities? ' 



What recoimiendations has the SC6 
presented to the school board? 



How did the board respond? 



Has the school board ever asked the SCGfor help? YES NO 

If so, what was the nature of their request? • ' , 




Vlease rate the degree to which.you agree or disagree with these ; 
statements: * - . ' ' 

a: The SC6 has-been able to resolve disagreements* and make decisions 
without- excessive use of time^ of l,oss of membership. 

Strongly "agree • 5 4 , 3< . Z 1^ —strongly dfsagree 



Comment: \; - 



b. ^ The .membership of the SCG has'^eft^tned constant since the group 
wa«; formed. ' 



was formed. 

<;+rnnniv aarPP • 5 4 3' YA\V^ -Str,6jigly di sagree' 



:c. The'time arid effort I've spent on SCG activities has been very 
worthwhile. 



Strortsily. agree . 5 4 , 3 2 "1 Strongly disagree 
Comment: 



d. Community people, students and staff are kept well informed 
about SCG activities. 

-Strongly agree 5 4 3 2 1- Strongly disagree 
Comment: 



a. Prior to becoming a member of the SCG I had: 



Little or no voice vj^; A great deal of influence 

in school decisions 1 2 1 ""4 5 on school decisions 

b. Now I have: 

' i tittle or no voice A great deal of influence 

v: in school decisions 1 2 3 4 5 on school decisions 

If there was a change tetween ratings for a and b, do you feel 

' it was due to, the process? YES NO ' ^ 



Please explain: 



I 

c. Given niy present level of 'influence on school decisions, I • 
would, prefer: MORE LESS SAME 

The rest of th'e staff, students and community members were kept 
informed about SCG -activities by the following me.thod(s): (Check 

all that '^pply) \ 

a. ;^ newspaper articles ^ 

.b.- word of mou|;h - .1 

c. \ radio announcements 

cL flyers/newsletters 

e. 'the "community notebook" 

f. other: 



1 
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' T3< " I have Id^trned-new skills &s'a result of participating in the 
, School- Cofiinunity Group:- • * - % , ^ 

'Strongly disagree 1 2 3 4* 5 /. Strongly agree 

Wljat do yoj; feel. you . have' learned? / " *• • 



'I r 



14, What changes, do you think would make the SC6 more effective? 



' 15,. What -personal benefits do you feel you have^received from being a ^ 
member of the,SC6 (i.e;, ^a chance to sVe friends more often, learned 
more* about the present school structure)? " 



*16. Please give any other comments about the SC6, positive or negative, 
which you feel are important in the space below: 




